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Abstract 

Background: Improvements in the quality of health care services are often measured using data present in medical 
records. Electronic Medical Records (EMRs) contain potentially valuable new sources of health data. However, data 
quality in EMRs may not be optimal and should be assessed. Data reliability (are the same data elements being 
measured over time?) is a prerequisite for data validity (are the data accurate?). Our objective was to measure the 
reliability of data for preventive services in primary care EMRs during the transition to EMR. 

Methods: Our data sources were randomly selected eligible patients' medical records and data obtained from 
provincial administrative datasets. Eighteen community-based family physicians in Toronto, Ontario that 
implemented EMRs starting in 2006 participated in this study. We measured the proportion of patients eligible for a 
service (Pap smear, screening mammogram or influenza vaccination) that received the service. We compared the 
change in rates of selected preventive services calculated from the medical record audits with the change in 
administrative datasets. 

Results: In the first year of EMR use (2006) services decreased by 8.7% more (95% CI -1 1.0%- -6.4%, p< 0.0001) 
when measured through medical record audits as compared with administrative datasets. Services increased by 
2.4% more (95% CI 0%-4.9%, p = 0.05) in the medical record audits during the second year of EMR use (2007). 

Conclusion: There were differences between the change measured through medical record audits and 
administrative datasets. Problems could include difficulties with organizing new data entry processes as well as 
continued use of both paper and EMRs. Data extracted from EMRs had limited reliability during the initial phase of 
EMR implementation. Unreliable data interferes with the ability to measure and improve health care quality 

Keywords: Measurement, Reliability, Validity, Electronic medical records, Preventive health services, Quality of health 
care, Primary medical care 



Background 

The quality, accuracy and completeness of the informa- 
tion in medical records is fundamental to good patient 
care and to quality improvement activities: "you cannot 
improve what you cannot measure" [1]. The transition 
from paper-based records to Electronic Medical Records 
(EMRs) has led to expectations that electronic health 
care data can and will be used to measure and improve 
the quality of care provided to patients [2,3]. 

However, the quality of the data entered in EMRs as part 
of routine care needs to be assessed; missing or inconsist- 
ent data may make the measurement of quality 
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problematic [4]. Many physicians continue to simultan- 
eously use both paper and electronic medical records, or 
"hybrid" charts, [5] leading to uncertainty as to where data 
reside. Problems already encountered in both Canadian 
and international primary care EMR settings include in- 
consistent or missing diagnostic coding and risk factor 
designation, "dirty data" (misspelled words, inconsistent 
word strings, free text strings instead of structured data), 
missing "metadata" or description of data content (for ex- 
ample, referral to "Dr Smith", where physician specialty is 
not listed) and data entered in inconsistent or incorrect 
database fields [4,6-13]. 

Data quality factors can be categorized as data com- 
pleteness (are all the data present?), data reliability (are 
data recorded in the same way across practices and over 
time?) and data validity (are the data correct?) [14,15]. 
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In a recent literature review of EMR data use for 
health outcome research, only 24% of studies had a data 
validation component, and most studies used paper 
based records as the gold standard [16]. The least com- 
mon method of validation was the use of administrative 
data [16,17]. The majority of data validation studies were 
not conducted in the primary care setting [16]. There 
were difficulties with the reference standards used to 
evaluate EMR data quality; some studies relied on unval- 
idated standards (surveys, interviews); current "gold 
standards" such as paper notes may be problematic 
[14,18,19]. As one editorial noted, "there are no agreed 
reference standards for reporting data quality in primary 
care and this limits measurement of data quality in elec- 
tronic patient records" [20]. While there are many pos- 
sible ways to measure EMR data quality and many areas 
that can be measured, [8,21,22] systematic reviews of 
data quality assessment have noted a focus on diagnostic 
data, laboratory testing, risk factors and demographic in- 
formation, with limited information on data quality 
regarding preventive services [7,15]. 

We recently studied the effects of the first two years of 
EMR implementation in the practices of eighteen 
community-based family physicians in Toronto [23]. We 
measured preventive services targeted by Ontario's pay for 
performance program. The pay-for-performance targets 
were based upon the percentage of a physicians eligible 
enrolled patients being provided with Pap smears, mam- 
mograms, influenza vaccinations, fecal occult blood 
screening and primary vaccinations of children under two 
within a specified timeframe [24-26]. We found no differ- 
ence in the change of service provision between physicians 
implementing EMR and a group who continued to use 
paper records [23]. 

As part of the study, we evaluated aspects of the quality 
of the data present for those preventive services in the 
medical records by comparing results obtained from med- 
ical record audits against external data sources. The exter- 
nal sources were provincial administrative claims-based 
data housed at the Institute for Clinical Evaluative Sciences 
(ICES). ICES is an independent, not-for-profit health ser- 
vices research organization funded by the Ministry of 
Health and Long Term Care of Ontario. Ontario has 
population-based coverage for eligible physician, labora- 
tory, diagnostic imaging and hospital services through the 
Ontario Health Insurance Plan. A copy of fees submitted 
to the Ministry of Health and Long Term Care of Ontario 
is maintained in anonymized administrative databases at 
ICES. 

As noted above, medical record reviews and audits have 
traditionally been considered to be the "gold standard" for 
certain health care services, and have been used to validate 
administrative data [27,28]. The administrative data we 
used to determine the rates of preventive services have not 



been validated using this method. However, we assumed 
that these administrative data captured a relatively con- 
stant proportion of services provided over the time during 
the period we studied. For example, if 75% of influenza 
vaccinations were available in administrative data in 2006, 
we would expect that approximately 75% would be avail- 
able in 2007. We know of no positive or negative changes 
during the time period studied that would have affected 
billing proportions captured in administrative databases. 
We therefore used the change in preventive services 
reported in the administrative data for the study practices 
as the reference standard and compared this change to 
results from the chart audits. This gave us a method of 
measuring the reliability of data recorded in EMRs over 
time as compared with administrative data. 

Our null hypothesis was that data for preventive ser- 
vices are reliably entered in clinical records over time 
during EMR implementation; there are no statistically 
significant or clinically important differences between 
changes in EMR data and changes in administrative data. 

The research question was: does the change in pre- 
ventive services in medical records before and during the 
first two years of EMR use agree with the change in pre- 
ventive services in administrative data? 

Methods 

Participants 

The eighteen physicians in this study had previously par- 
ticipated in a pay-for-performance study [29] and data 
on their characteristics and performance for 2004 and 
2005 were available. They changed to a blended capita- 
tion model at the end of 2004, in which patients formally 
register (or roster) with a family physician. These physi- 
cians were exposed to pay-for-performance for prevent- 
ive services in 2005 and began EMR implementation 
early in 2006. All participating physicians used the same 
EMR software (Nightingale On Demand®). We studied 
the change in preventive services in the two years prior 
to EMR implementation (2004 and 2005) and the first 
two years of EMR implementation (2006 and 2007). 

The principal investigator was also a participant in this 
study. 

Outcome measures 

The study end point was whether or not a preventive 
service was documented within the required time period 
for an eligible patient. The target patient population con- 
sisted of all eligible enrolled patients. Documentation 
that the patient received the service through another 
health care provider was acceptable. Information on ser- 
vices and exclusion criteria is presented in Table 1; chil- 
dren's vaccinations were not examined, as billing codes 
in administrative databases also include vaccinations 
other than the five used in the study. 
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Table 1 Eligibility criteria, exclusion criteria, and required period for preventive service provision 



Service 


Eligible population 


Exclusion criteria 


Required period for service provision* 


Pap smears 


Enrolled women age 35 to 69 


Previous hysterectomy 


Documented service within the past 
30 months prior to March 31st 


Screening mammograms 


Enrolled women age 50 to 69 


History of breast cancer 


Documented service within the past 
30 months prior to March 31st 


Influenza vaccination 


Enrolled patients age 65 or over 




Documented service from October 1st 
to December 31st of previous year 


Fecal occult blood test 


Enrolled patients age 50 to 74 


History of colorectal cancer; history 
of inflammatory bowel disease; 
colonoscopy within the past five 
years 


Documented service in the past 
30 months prior to March 31st 



*The fiscal year end in Ontario's health care system is March 31st. For example, March 31st 2007 would be considered the 2006 year end, and March 31st 2008 
would be considered the 2007 year end. 



The denominators were the number of patients eligible 
for each service who were rostered to the physicians in 
each cohort by March 31st of each fiscal year (for ex- 
ample, March 31st 2005 for the 2004 fiscal year). Physi- 
cians report the preventive service performance levels 
they have achieved to the Ministry of Health and Long 
Term Care on March 31st 

The numerators were the number of eligible rostered 
patients who received a service in the 30 months prior 
to March 31st of each year for Pap smears, mammo- 
grams or fecal occult blood testing, or an influenza vac- 
cination from October 1st to December 31st of the prior 
year. The rate of service was defined as the proportion of 
eligible patients receiving a service at least once in the 
previous 30 months (Pap smears, mammograms, fecal 
occult blood tests) or in the previous fall (October 1st to 
December 31st) for influenza vaccination. 

We calculated a composite process score [30]. This is 
calculated by using the total number of medical records 
audited for eligible patients for each physician as the de- 
nominator, and the total number of services recorded in 
the audits as the numerator. We did not obtain fecal oc- 
cult blood testing for 2004 as part of our previous study 
on pay for performance; fecal occult blood testing was 
not part of the Ontario pay for performance program 
until 2006. This measure was not included in the com- 
posite score due to lack of complete data. The composite 
process score therefore included mammography, Pap 
smears and influenza vaccinations from 2004 to 2007. 

Data sources 
Chart audits 

We primarily audited the electronic medical records. 
However, when data were unavailable in an EMR, we 
retrieved data from the paper chart. We determined that 
40 charts per service per provider would be required to 
achieve a study power of 80% to detect a clinically im- 
portant increase in service provision of 5% or higher, 
with an alpha level of .05. To further enhance statistical 



power, we audited 50 charts per year, per service, per 
physician. 

Five data auditors abstracted data. The research coordin- 
ator initially audited ten charts for each service in two 
practices and reviewed this with the principal investigator. 
The coordinator then trained each data auditor, and 
reviewed at least ten charts for each service. The data were 
independently entered in an Epi Info database [31] by two 
data entry clerks. Each clerk entered a training sample of 
at least ten charts for each service. A randomly selected 
10% sample of data for each service, each year, and each 
physician was re-audited and entered in the database; we 
used the Kappa statistic to compare the two audits. 

Administrative audits 

We obtained administrative data for the entire practices 
from ICES, using the following datasets: 

• The ICES Physician Database (IPDB) for information 
on physician country of graduation; 

• The Corporate Provider Database (CPDB) for 
information on physicians' Ontario Health Insurance 
Plan billing number; 

• The Ontario's Registered Persons Database (RPDB) for 
patient age (as of August 31st 2007), gender and 
immigration recency by date of OHIP registration [32]; 

• The Client Agency Program Enrolment (CAPE) 
tables for information on patient enrolment in each 
physicians roster; 

• Statistics Canada data on neighborhood income, 
linked to patients' residential postal code for 
estimates of income quintiles [32]; 

• The Canadian Institute for Health Information's 
Discharge Abstract Database for hospital discharge 
diagnoses; 

• The Ontario Health Insurance Plan for billing and 
diagnostic data to identify patient visits and diagnoses; 

• The Ontario Diabetes Database (ODD) for diabetics 
[28]; 
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• The Ontario Asthma Database (OASIS) [33]; 

• The Ontario Congestive Heart Failure Database [34]; 

• The Ontario Chronic Obstructive Pulmonary 
Disease Database [35]; 

• The Ontario Hypertension Database [27]; 

• The Ontario Myocardial Infarction Database [36]; 

• The Ontario Cancer registry for information on 
breast or colorectal cancer 

• The Ontario Breast Screening Program (OBSP) for 
information on mammography 

A detailed description of the billing codes for the ad- 
ministrative data is provided in Additional file 1. 

Analysis 

We first calculated the composite process score for each 
year, for both medical record audits and administrative 
data. An equal number of charts had been randomly 
audited for each service in the medical record; we there- 
fore assigned an equal weight to each service in the ad- 
ministrative dataset (which contained data for the entire 
practice) for the composite score calculation. We then 
adjusted for differences in patient age [37] using logistic 
regression. We used the Generalized Estimating Equa- 
tion to adjust for the clustering structure of the data in 
regression models. 

Next, we compared the composite process score found 
in medical record audits and administrative data for each 
year using the chi-square test. We then compared the 
year over year change in the composite process score 
(that is, the percent of change found in the medical rec- 
ord audits and the percent of change found in the ad- 
ministrative data) using logistic regression. 

A difference of 5% in the change of services between 
medical record audits and administrative data was con- 
sidered clinically important. 

Analyses were performed with the use of SAS software, 
version 9.2 (SAS Institute). All tests were two sided and 
p values less than 0.05 were considered statistically 
significant. 



The study was approved by the University of Toronto's 
Research Ethics Board; the Sunnybrook Research Ethics 
Board approved the use of ICES data. All physicians pro- 
vided written informed consent. 

Results 

Physician characteristics and patient characteristics are 
presented in Table 2 and 3, respectively. 

A comparison of the composite scores obtained from 
the medical record audits and from administrative data 
is presented in Table 4. There was a statistically signifi- 
cant greater increase of 4.2% (95% CI 2.0%-6.4%) in the 
change in services found in medical record audits follow- 
ing the introduction of pay for performance in 2005. 
However, this did not reach the previously identified 
clinically important level of 5%. There was a statistically 
significant and clinically important larger decrease in ser- 
vices in the first year of EMR when these services were 
measured using medical record audits rather than ad- 
ministrative data. Measured services declined by 8.7% 
(95% CI -11.0% 6.4%) more when measured by med- 
ical record audits. There was no statistically significant 
or clinically important difference in change in the follow- 
ing year 

There was a significantly smaller proportion of services 
found in the medical record audits as compared with the 
administrative data in the year that the EMR was intro- 
duced (2006). There were more services found in the 
medical record audits for the two years prior to EMR 
(2004 and 2005), and no difference was found in the fol- 
lowing year (2007). 

Table 5 presents data for individual services derived 
from medical record audits and administrative data. 
These are presented graphically in Figure 1 (for Pap 
smears and mammograms) and Figure 2 (for influenza 
vaccinations and fecal occult blood tests). 

The intracluster correlation for each service in medical 
record audits was generally small, at 0.036 for influenza 
vaccination, 0.0197 for fecal occult blood testing, 0.0189 
for mammography, and 0.009 for Pap smears. Increasing 



Table 2 Characteristics of physicians 



Variable 




Physicians (N = 18) 


Year of graduation* 


Median (range) 


1977(1964-1992) 


Gender* 


Male (%) 


10(56) 


CCFP* 


N (%) 


11(61) 


Number of MDs in practice* 


Median (range) 


3(1 to 6) 


Number of hours worked per week* 


Median (range) 


42(30 to 60) 


Number of patients per physician* 


Median (range) 


1206(630-2200) 


Canadian vs foreign graduate! 




16/18 



Note: CCFP Certificant of the College of Family Physicians of Canada. 

^Obtained from self reports. 

tObtained from administrative databases. 
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Table 3 Characteristics of the total practice population 

Variable* 



Patients 


N (Mean) 


23,514 (1,306) 


Age as of August 31st 2007 


Median (IQR) 


45 (27-60) 


Patient Gender 


Male (%) 


10,106 (43.0) 


Neighbourhood income quintile [32] 






Unknown 


N (%) 


51 (0.2) 


1 (lowest) 


N (%) 


3,084 (13.1) 


2 


N (%) 


3,643 (15.5) 


3 


N (%) 


4,345 (18.5) 


4 


N (%) 


5,091 (21.7) 


5 (highest) 


N (%) 


7,300 (31.0) 


Recent immigrant [32] 




1,398 (5.9) 


Comprehensiveness of care + [32,38] 


Mean±SD 


0.54 ±0.35 


Overall morbidity (Resource Utilization Bands)* [39,40] 


Mean±SD 


2.73 ± 1 .02 


0 


N (%) 


1,047 (4.5) 


1 


N (%) 


1,480 (6.3) 


2 


N (%) 


4,778 (20.3) 


3 


N (%) 


12,567 (53.4) 


4 


N (%) 


2,783 (11.8) 


5 


N (%) 


859 (3.7) 


Overall comorbidity (Adjusted Diagnosis Groups) § [39,40] 


Mean±SD 


4.77 ±3.04 


0 


N (%) 


1,046 (4.4) 


1-4 


N (%) 


11,189 (47.6) 


5-9 


N (%) 


9,502 (40.4) 


10+ 


N (%) 


1,777 (7.6) 


Diabetes [28] 


N (%) 


1,934 (8.2) 


Congestive heart failure [34] 


N (%) 


386 (1.6) 


Hypertension [27] 


N (%) 


5,594 (23.8) 


Myocardial infarct [36] 


N (%) 


311 (1.3) 


Asthma [33] 


N (%) 


3,143 (13.4) 


Chronic obstructive pulmonary disease [35] 


N (%) 


1,120 (4.8) 


Mental health [41] 


N (%) 


4,937 (21.0) 



Note: IQR interquartile range; SD standard deviation. 

* Obtained from administrative databases. 

+ Comprehensiveness of care was determined by measuring the percentage of bills for 21 commonly provided services that were provided by the patient's own 
family physician. 

* Resource utilization bands indicate morbidity and expected health care system use, from 0 (lowest) to 5 (highest). 

§ Adjusted diagnosis groups indicate comorbidity, from 0 groups (lowest level of comorbidity) to 10+ groups (highest level). 



patient age was a significant factor for the provision of a 
preventive service (data not shown). Using medical rec- 
ord audits, there was a decrease in the provision of 
mammograms, pap smears and influenza vaccinations 
between 2005 and 2006 (p < 0.001); there was no signifi- 
cant change in fecal occult blood testing (p - 0.12). 

Administrative data showed no significant change be- 
tween 2005 and 2006 for Pap smears (^ = 0.52) or mam- 
mography (/? = 0.48); there was a decrease in influenza 



vaccination (p< 0.0001) and an increase in fecal occult 
blood testing (p < 0.0001). 

The overall agreement between the sample of medical 
records that were re-audited was acceptable (kappa 0.954). 

Discussion 

We present a novel method of assessing an aspect of 
EMR data quality. We compared the rate of change of 
services over time in the EMR with the rate of change in 
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Table 4 Comparison of medical record audits and administrative data, composite score for mammography, Pap smears 
and influenza vaccinations 



Year 


Patients 

receiving service, 
medical record 
audits, n/N, 
% (95% CI)* 




Patients receiving 
service, 
administrative 
data, n/N, 
% (95% CI) 




Difference 
between 
medical record 
audits and 
administrative 
data within 
year (95% CI) 


Difference in the 
change from 
previous year 
(95% CI) 


2004 


2264/2807,80.7%(79.2%- 


-82.1%) 


1 1074/1 4096,78.6%(77.9%- 


-79.2%) 


+2.1% (0.596-3.7%) 




2005 (post pay-for- 
performance) 


2385/2809,84.9%(83.6%- 


-86.2%) 


11 735/1 4927,78.6%(78.0%- 


-79.3%) 


+6.3% (4.8%-7.8%) 


4.2% greater increase 

with chart audits (2.0%-6.4%) 


2006 (first year 

after EMR introduction) 


1 995/2696,74.1 %(72.4%- 


-75.7%) 


1 1 692/1 5291 ^e.So/oCyS.So/o- 


-77.1%) 


-2.4% (-4.2%- -0.6%) 


8.7% greater decrease 
with chart audits 
(-11.0%- -6.4%) 


2007 (second year 
after EMR introduction) 


2076/2703,77.2%(75.6%- 


-78.8%) 


1 2005/1 5559,77.2%(76.5%- 


-77.8%) 


0 (-1.7%- 1.8%) 


2.4% greater increase 
with chart audits(0%-4.9%) 



Percentages are adjusted for patient age. 

administrative databases. We found a greater decrease in 
preventive services in the first year of EMR when data 
were obtained from medical record audits as opposed to 
administrative data sources. The discrepancy between 
the two methods points to the possibility of missing data 
for preventive services in the medical records. 

Reliability reflects data stability, and is necessary (but 
not sufficient) for data validity [15]. Our administrative 
data for Pap smears were based on billing codes submit- 
ted by laboratories and physician billing codes as detailed 
in Additional file 1. Laboratory billings would not have 
been affected by EMR implementation. Administrative 
data for mammograms were based on radiology billing 
codes and data from the Ontario Breast Screening Pro- 
gram, as shown in Additional file 1. These also were in- 
dependent of EMR implementation. 

The change in influenza vaccinations was similar in 
the medical record audits and in the administrative data, 
perhaps reflecting fewer problems with documentation. 
Documenting an influenza vaccination does not require 
looking two years back for the presence of the service, as 
mammography or Pap smears do; therefore, there may 
be a less complex workflow associated with recording 
this service during the move to EMR. There were delays 



in vaccine delivery in 2006 and 2007, which could ac- 
count for the lower levels of vaccination found in both 
the medical record audits and the administrative data 
during the fall season of those two years. 

Challenges with the data needed to measure quality have 
been reported elsewhere in the literature [7,8,11,14,15]. 
Roth [42] found that only one-third of the indicators 
needed for a quality assessment program could be easily 
extracted from EMRs, and that there were difficulties asso- 
ciated with provider data entry habits and differences 
across different EMR applications [42]. The structure of 
the EMR is more complex than that of the paper chart: 
physicians may not be entering data in consistent or 
expected locations, making it difficult to extract [42]. Phy- 
sicians and auditors may have challenges in navigating the 
electronic medical record. Data from external sources may 
be scanned in and may not be extractable electronically 
[43] . Physicians may continue to use both paper and elec- 
tronic medical records, [5] scattering data across two dif- 
ferent systems and possibly increasing the amount of 
incomplete or duplicated data in audits. Research and 
quality improvement projects using EMR data will need to 
consider the quality of data entered in the EMR, as well as 
issues specific to the EMR application used [42]. 



Table 5 Individual services levels derived from medical records audits and from administrative data 



Service 




Source 


2004 


2005 


2006 


2007 


Fecal occultblood tests, 0 / 


* (n/N) 


Administrativedata 


21.7% (5139/1117) 


23.2% (1216/5239) 


23.6% (1232/5216) 


26.4% (1355/5137) 


Fecal occultblood tests, 0 / 


* (n/N) 


Medical records 




27.196(236/871) 


28.7%(250/870) 


32.1%(285/888) 


Influenza vaccinations,% 


(n/N) 


Administrativedata 


74.2% (3163/4263) 


69.5% (3095/4453) 


62.6% (2880/4601) 


64.3% (3072/4776) 


Influenza vaccinations,% (n/N) 


Medical records 


76.2%(745/978) 


83.2%(790/949) 


70.7%(638/902) 


69.896(621/901) 


Mammograms,% (n/N) 




Administrativedata 


79.3% (2671/3367) 


81.3% (3095/4453) 


82.0% (3026/3692) 


82.8% (3072/4776) 


Mammograms,% (n/N) 




Medical records 


81.9%(751/917) 


85.4%(791/926) 


75.2%(672/894) 


80.9%(728/900) 


Pap smears,% (n/N) 




Administrativedata 


81.0% (5240/6466) 


83.1% (5736/6903) 


82.7% (5786/6998) 


82.9% (5807/7007) 


Pap smears,% (n/N) 




Medical records 


84.2%(768/912) 


86.1%(804/934) 


76.1%(685/900) 


79.7%(7 19/902) 
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90% 



85% 



80% 



75% 



70% 




A Mammograms (medical 
records) 

— * — Pap smears (medical 
records) 

-•^•-Mammograms 

(administrativedata) 

Pap smears (administrative 
data) 



2004 2005 2006 2007 

Figure 1 Mammogram and Pap smear service levels derived from medical record audits and from administrative data. 



Baron described the implementation of a mammography 
recall program within an innovative, fully computerized 
primary care group practice [43]. The system was initially 
unable to properly audit mammograms and to produce ac- 
curate lists of patients to be recalled; mammograms were 
scanned in but were not recognized by the EMR. Baron 
described the development and implementation of practice 
processes to "tag" incoming mammograms so that patients 
could be properly categorized as having or not having had 
a mammogram within the previous two years [43] . Essen- 
tially, the practice cleaned and restructured their mam- 
mography data so that data were reliably entered in an 
area of the EMR where they could be audited. 

The quality of information (accuracy, reliability, complete- 
ness) has been found to be associated with empirical mea- 
sures of success in implementing information technology in 
the business literature [44]. Unreliable information makes a 
system less useful, impacting implementation efforts and 



decreasing the net benefits that could be obtained from the 
technology [45]. 

Measuring performance depends on accurate documenta- 
tion [1,46,47]. Once reliable and valid data have been 
entered into the EMR, interventions that have been found 
to increase performance, such as audits and feedback to 
clinicians, [48,49] point of care prompts for needed inter- 
ventions, [49-51] and reminder letters to patients [52,53] 
can be effectively implemented. We found a lack of im- 
provement in preventive service documentation associated 
with the early stages of EMR implementation [23]. It is pos- 
sible that elements of those negative results were due to 
problems with data quality during the early EMR imple- 
mentation efforts. 

Limitations 

This study was limited to a group of selected physicians 
in Toronto. However, all physicians in this study were 
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Figure 2 Influenza vaccination and fecal occult blood service levels derived from medical record audits and from administrative data. 
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practicing in community-based settings, similar to the 
majority of family physicians in Ontario [54]. We studied 
a single commercially available EMR application, and 
results may differ for different EMRs. Nonetheless, a re- 
cent review of data in a national primary care EMR data- 
base using nine different EMR applications found that 
data quality problems were pervasive across all platforms 
[4]. Administrative data and patient level data in the 
EMR could not be linked; we compared practice level 
data using randomly selected EMR charts. Nonetheless, 
there is no a priori reason to suspect that there are sys- 
tematic differences between the two samples. 

Conclusion 

In conclusion, we found that, in the early phase of EMR 
implementation, data for the preventive services we 
measured were not reliably entered over time in the 
medical records we audited when compared to provin- 
cial administrative data. Data reliability should be 
assessed if EMR-based data are used to measure and im- 
prove quality. 

Additional file 



Additional filel: Inclusion and exclusion criteria for administrative 
cohorts. 



Competing interests 

The authors declare that they have no competing interests. 
Authors' contributions 

All authors contributed to the concept and design of the study. MG and JB 
contributed to the data gathering. All authors contributed to the analysis, 
interpretation and preparation of the manuscript. All authors read and 
approved the final manuscript. 

Acknowledgements 

This study was funded by a research grant from the Ontario Ministry of 
Health and Long Term Care Health System Strategy Division (Grant No 
020709). 

This study was also supported by the Institute for Clinical Evaluative Sciences 
(ICES), which is funded by an annual grant from the Ontario Ministry of 
Health and Long-Term Care (MOHLTC). The opinions, results and conclusions 
reported in this paper are those of the authors and are independent from 
the funding sources. No endorsement by ICES or the Ontario MOHLTC is 
intended or should be inferred. 

Author details 

department of Family and Community Medicine, University of Toronto, 
Duncan Mill Road, Suite 705, Toronto, ON, Canada, department of Health 
Policy, Management & Evaluation, University of Toronto, Institute for Clinical 
Evaluative Sciences, Toronto, Canada, department of Family and Community 
Medicine, Institute for Clinical Evaluative Sciences, Centre for Research on 
Inner City Health, St. Michael's Hospital, University of Toronto, Toronto, 
Canada. 4 Dalla Lana School of Public Health, University of Toronto, Toronto, 
Canada, department of Family and Community Medicine, University of 
Toronto, Institute for Clinical Evaluative Sciences, Dalla Lana School of Public 
Health, Toronto, Canada. 

Received: 10 August 201 1 Accepted: 14 February 2012 
Published: 14 May 2012 



References 

1 . Newcomer LN: Perspective: physician, measure thyself. Health Aff 1 998, 1 7 
(4)32-35. 

2. Romanow R: Building on values: the future of health care in Canada. Ottawa: 
National Library of Canada; 2002. 

3. Have paper records passed their expiry date? CMAJ 2005, 173(7):725. 

4. CIHI CPCSSN data analysis report: cycle 2A. Toronto: Canadian Institutes for 
Health Information; 2010. 

5. National Physician Survey, [http://www.nationalphysiciansurvey.ca/nps/ 
2007_S u rvey/2007 res u Its-e.as p] . 

6. Birtwhistle R, Keshavjee K, Lambert-Lanning A, Godwin M, Greiver M, Manca 
D, Lagace C: Building a pan-Canadian primary care sentinel surveillance 
network: initial development and moving forward. J Am Board Fam Med 
2009, 22(4)412-422. 

7. Brouwer H, Bindels P, Weert H: Data quality improvement in general 
practice. Fam Pract 2006, 23(5):529-536. 

8. Whitelaw FG, Nevin SL, Milne RM, Taylor RJ, Taylor MW, Watt AH: 
Completeness and accuracy of morbidity and repeat prescribing records 
held on general practice computers in Scotland. Br J Gen Pract 1996, 46 
(404):181-186. 

9. Selak V, Wells S, Whittaker R, Stewart A: Smoking status recording in GP 
electronic records: the unrealised potential. Inform Prim Care 2006, 14 
(4):235-241. discussion 242-235. 

10. Mant J, Murphy M, Rose P, Vessey M: The accuracy of general practitioner 
records of smoking and alcohol use: comparison with patient 
questionnaires. J Pub Health Med 2000, 22(2):1 98-201. 

1 1 . Hollowell J: The General Practice Research Database: quality of morbidity 
data. Popul Trends 1 997, 87:36-40. 

1 2. Mukhi S, Keshavjee K: Developing a primary care electronic medical record 
chronic disease surveillance system in Canada: data quality and lessons learned. 
Calgary: In Canadian Association of Health Services and Policy Research; 2009. 

13. Birtwhistle R, Keshavjee K, Martin K: Improving data quality in EMRs for 
chronic disease management. In Family Medicine Forum. Calgary: Lambert- 
Lanning A; 2009. 

14. Chan KS, Fowles JB, Weiner JP: Review: electronic health records and the 
reliability and validity of quality measures: a review of the literature. Med 
Care Res Rev 2010, 67(5):503-527. 

15. Thiru K, Hassey A, Sullivan F: Systematic review of scope and quality of 
electronic patient record data in primary care. BMJ 2003, 326(7398):1070. 

16. Dean BB, Lam J, Natoli JL, Butler Q, Aguilar D, Nordyke RJ: Review: use of 
electronic medical records for health outcomes research: a literature 
review. Med Care Res Rev 2009, 66(6):61 1-638. 

1 7. Benin AL, Vitkauskas G, Thornquist E, Shapiro ED, Concato J, Asian M, 
Krumholz HM: Validity of using an electronic medical record for assessing 
quality of care in an outpatient setting. Med Care 2005, 43(7):691-698. 

18. Tang PC, LaRosa MP, Gorden SM: Use of computer-based records, 
completeness of documentation, and appropriateness of documented 
clinical decisions. J Am Med Inform Assoc 1999, 6(3):245-251. 

19. Jordan K, Porcheret M, Croft P: Quality of morbidity coding in general 
practice computerized medical records: a systematic review. Fam Pract 
2004, 21(4)396-412. 

20. Majeed A, Car J, Sheikh A: Accuracy and completeness of electronic 
patient records in primary care. Fam Pract 2008, 25(4):21 3-214. 

21. Jick H, Jick SS, Derby LE: Validation of information recorded on general 
practitioner based computerised data resource in the United Kingdom. 
8M7 1991,302(6779):766-768. 

22. Porcheret M, Hughes R, Evans D, Jordan K, Whitehurst T, Ogden H, Croft P, 
North Staffordshire General Practice Research N: Data quality of general 
practice electronic health records: the impact of a program of 
assessments, feedback, and training. J Am Med Inform Assoc 2004, 
11(1):78-86. 

23. Greiver M, Barnsley J, Glazier R, Harvey B, Moineddin R: Effect of 
electronic medical record implementation on the provision of 
preventive services in a pay for performance environment. Canadian 
Family Physician 201 1, 57:e390-e397. 

24. Family health network agreement: General blended payment template. 
Ministry of Health and Long Term Care of Ontario and the Ontario Family 
Health Network Corporation; 2002. 

25. Memorandum of agreement between the Ontario Medical Association and the 
Ontario Ministry of Health and Long Term Care. Toronto: Government of 
Ontario; 2004. 



Greiver et al. BMC Health Services Research 201 2, 1 2:11 6 Page 9 of 9 

http://www.biomedcentral.eom/1 472-6963/1 2/1 1 6 



26. Memorandum of agreement between the Ontario Medical Association and the 
Ontario Ministry of Health and Long Term Care. Toronto: Government of 
Ontario; 2003. 

27. Tu K, Campbell NR, Chen ZL, Cauch-Dudek KJ, McAlister FA: Accuracy of 
administrative databases in identifying patients with hypertension. Open 
Med 2007, 1(1):e18-e26. 

28. Hux JE, Ivis F, Flintoft V, Bica A: Diabetes in Ontario: determination of 
prevalence and incidence using a validated administrative data 
algorithm. Diabetes Care 2002, 25(3)512-516. 

29. Greiver M, Kasperski J, Barnsley J, Rachlis V: Measuring quality improvements 
in preventive care services in the first two family health networks in the greater 
Toronto area. Toronto: Ontario College of Family Physicians; 2006. 

30. Lindenauer PK, Remus D, Roman S, Rothberg MB, Benjamin EM, Ma A, 
Bratzler DW: Public reporting and pay for performance in hospital quality 
improvement. N Engl J Med 2007, 356(5):486-496. 

31. Epi Info. 332nd edition. Atlanta: Centers for Disease Control and Prevention; 
2005. 

32. Glazier RH, Klein-Geltink J, Kopp A, Sibley LM: Capitation and enhanced 
fee-for-service models for primary care reform: a population-based 
evaluation. CMAJ 2009, 180(1 1):E72-E81. 

33. Gershon AS, Wang C, Guan J, Vas i levs ka-Ri stovs ka J, Cicutto L, To T: 
Identifying patients with physician-diagnosed asthma in health 
administrative databases. Can Respir J 2009, 1 6(6):1 83-1 88. 

34. Ko DT, Alter DA, Austin PC, You JJ, Lee DS, Qiu F, Stukel TA, Tu JV: Life 
expectancy after an index hospitalization for patients with heart failure: 
a population-based study. Am Heart J 2008, 1 55(2)324-331 . 

35. Gershon AS, Wang C, Guan J, Vas i levs ka-Ri stovs ka J, Cicutto L, To T: 
Identifying individuals with physcian diagnosed COPD in health 
administrative databases. COPD 2009, 6(5)388-394. 

36. Jackevicius CA, Li P, Tu JV: Prevalence, predictors, and outcomes of 
primary nonadherence after acute myocardial infarction. Circulation 2008, 
117(8):1 028-1 036. 

37. Asch SM, Kerr EA, Keesey J, Adams JL, Setodji CM, Malik S, McGlynn EA: Who 
is at greatest risk for receiving poor-quality health care? N Engl J Med 
2006, 354(1 1):1 147-1 156. 

38. Shortell SM: Continuity of medical care: conceptualization and 
measurement. Med Care 1976, 14(5)377-391. 

39. The Johns Hopkins University ACG Case-Mix Adjustment System: Baltimore. 
Md: Johns Hopkins University School of Hygiene and Public Health; 1997. 

40. The Johns Hopkins University ACG Case-Mix System. [http://www.acg.jhsph. 
edu/html/HowDoACGsWorkhtm]. 

41. Steele LS, Glazier RH, Lin E, Evans M: Using administrative data to measure 
ambulatory mental health service provision in primary care. Med Care 
2004, 42(10):960-965. 

42. Roth CP, Lim YW, Pevnick JM, Asch SM, McGlynn EA: The challenge of 
measuring quality of care from the electronic health record. Am J Med 
Qual 2009, 24(5)385-394. 

43. Baron RJ: Quality improvement with an electronic health record: 
achievable, but not automatic. Ann Intern Med 2007, 147(8)349-552. 

44. DeLone WH, McLean ER: Information systems success: the quest for the 
dependent variable. Inform Syst Res 1992, 3(1):60-95. 

45. Delone WH, McLean ER: The DeLone and McLean model of information 
systems success: a ten-year update. J Manage InfSyst 2003, 19(4):9-30. 

46. Chassin MR, Galvin RW, The National Roundtable on Health Care Quality: The 
urgent need to improve health care quality: Institute of Medicine National 
Roundtable on Health Care Quality. JAMA 1 998, 280(1 1):1 000-1 005. 

47. Casalino L, Gillies RR, Shortell SM, Schmittdiel JA, Bodenheimer T, Robinson 
JC, Rundall T, Oswald N, Schauffler H, Wang MC: External incentives, 
information technology, and organized processes to improve health care 
quality for patients with chronic diseases. Jama 2003, 289(4):434-441. 

48. Perlin JB, Kolodner RM, Roswell RH: The Veterans Health Administration: 
quality, value, accountability, and information as transforming strategies 
for patient-centered care. Am J Manag Care 2004, 10(1 1 Pt 2):828-836. 

49. Craig TJ, Perlin JB, Fleming BB: Self-reported performance improvement 
strategies of highly successful veterans health administration facilities. 
Am J Med Qual 2007, 22(6):438-444. 

50. Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD: 
Improving preventive care by prompting physicians. Arch Intern Med 
2000, 160(3)301-308. 

51. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, 
Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision 



52. 



53. 



54. 



support systems on practitioner performance and patient outcomes: a 

systematic review. JAMA 2005, 293(1 0):1 223-1 238. 

Szilagyi PG, Bordley C, Vann JC, Chelminski A, Kraus RM, Margolis PA, 

Rodewald LE: Effect of patient reminder/recall interventions on 

immunization rates: a review. Jama 2000, 284(1 4):1 820-1 827. 

Stone EG, Morton SC, Hulscher ME, Maglione MA, Roth EA, Grimshaw JM, 

Mittman BS, Rubenstein LV, Rubenstein LZ, Shekelle PG: Interventions that 

increase use of adult immunization and cancer screening services: a 

meta-analysis. Ann Intern Med 2002, 136(9)641-651. 

Schultz SETJ, Guttmann A, Jaakkimainen L: In Characteristics of primary care 

practice, Primary care in Ontario: ICES Atlas. Edited by Jaakkimainen LUR, 

Klein-Geltink JE, Leong A, Maaten S, Schultz SE, Wang L. Toronto: Institute 

for Clinical Evaluative Sciences; 2006. 



doi:1 0.1 1 86/1 472-6963-1 2-1 1 6 

Cite this article as: Greiver et al: Measuring data reliability for preventive 
services in electronic medical records. BMC Health Services Research 2012 
12:116. 



Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 



Submit your manuscript at 
www.biomedcentral.com/submit 



(3 BioMed Central 



